Camera Aided Motion Direction and Speed Estimation

ABSTRACT

Methods and devices for camera aided motion direction and speed estimation are disclosed. The method of determining position characteristics of a mobile device comprises capturing a plurality of images that represent views from the mobile device, adjusting perspectives of the plurality of images based at least in part on an orientation of the mobile device, determining a misalignment angle with respect to a direction of motion of the mobile device using the plurality of images, and storing the misalignment angle and the direction of motion in a storage device.

FIELD

The present disclosure relates to the field of wireless communications.In particular, the present disclosure relates to determining positioncharacteristics of a mobile device.

BACKGROUND

Various mobile device applications, such as navigation aids, businessdirectories, local news and weather services, or the like, leverageknowledge of the position of the device. In various cases, the positionof a mobile device is identified via motion tracking with respect to thedevice. For example, in the case of sensor-aided pedestrian navigationapplications, motion direction is determined using the orientation ofthe device sensors in relation to the direction of forward motion. Theangle between the orientation of the mobile device and the forwardmotion direction is referred to as the alignment angle or misalignmentangle. For mobile applications that assist users while they travel onfoot, the orientation of the mobile device may change frequently, whichmay in turn change the misalignment angle of the mobile devicefrequently, and may adversely affect the user experience of such mobileapplications.

Therefore, there is a need for methods and systems that can address theabove issues of conventional solutions.

SUMMARY

The present disclosure relates to methods and apparatuses fordetermining position characteristics of a mobile device. According toembodiments of the present disclosure, a method of determining positioncharacteristics of a mobile device comprises capturing a plurality ofimages that represent views from the mobile device, adjustingperspectives of the plurality of images based at least in part on anorientation of the mobile device, determining a misalignment angle withrespect to a direction of motion of the mobile device using theplurality of images, and storing the misalignment angle and thedirection of motion in a storage device. The method further comprisesapplying the misalignment angle and a confidence of the misalignmentangle to navigate a user of the mobile device.

The method of adjusting perspectives of the plurality of imagescomprises at least one of: adjusting perspectives of the plurality ofimages based on the orientation of the mobile device calculated usingdata collected from one or more sensors; compensating for perspectivesof the plurality of images using an area near centers of the pluralityof images; and compensating for perspectives of the plurality of imagesbased at least in part on a weighted average of locations of features inthe plurality of images.

The method of determining the misalignment angle comprises trackingfeatures from the plurality of images, estimating direction of motion ofthe mobile device, estimating the orientation of the mobile device usingsensor data, and determining the misalignment angle based at least inpart on the direction of motion and the orientation of the mobiledevice. The method of tracking features from the plurality of imagescomprises rejecting outliers in features of the plurality of images toeliminate at least one moving object in the plurality of images.

The method further comprises at least one of: determining a confidenceof the misalignment angle with respect to the direction of motion of themobile device using information provided by a gyroscope of the mobiledevice, determining the confidence of the misalignment angle withrespect to the direction of motion of the mobile device usinginformation provided by a magnetometer of the mobile device, anddetermining the confidence of the misalignment angle with respect to thedirection of motion of the mobile device using features of the pluralityof images.

The method further comprises determining a speed estimation of themobile device, determining a confidence of the speed estimation of themobile device, and applying the speed estimation and the confidence ofthe speed estimation to navigate a user of the mobile device. The methodof determining a speed estimate comprises extracting features from theplurality of images, computing an average displacement of the mobiledevice using the features from the plurality of images, and computingthe speed estimate based at least in part on the average displacement ofthe mobile device. The method of computing the speed estimate comprisescomparing the features from the plurality of images, determining aseparation of pixels between two consecutive images in the plurality ofimages, determining a time interval between the two consecutive images,and calculating the speed estimate of the mobile device in accordancewith the separation of pixels and the time interval between the twoconsecutive images. The method further comprises calibrating a height ofthe mobile device according to at least one of GPS location informationand WIFI location information of the mobile device.

In yet another embodiment, an apparatus comprises a control unitincluding processing logic, where the processing logic comprises logicconfigured to capture a plurality of images that represent views fromthe mobile device, logic configured to adjust perspectives of theplurality of images based at least in part on an orientation of themobile device, logic configured to determine a misalignment angle withrespect to a direction of motion of the mobile device using theplurality of images, and logic configured to store the misalignmentangle and the direction of motion in a storage device.

In yet another embodiment, a computer program product comprisingnon-transitory medium storing instructions for execution by one or morecomputer systems, the instructions comprise instructions for capturing aplurality of images that represent views from the mobile device,instructions for adjusting perspectives of the plurality of images basedat least in part on an orientation of the mobile device, instructionsfor determining a misalignment angle with respect to a direction ofmotion of the mobile device using the plurality of images, andinstructions for storing the misalignment angle and the direction ofmotion in a storage device.

In yet another embodiment, a system comprises means for capturing aplurality of images that represent views from the mobile device, meansfor adjusting perspectives of the plurality of images based at least inpart on an orientation of the mobile device, means for determining amisalignment angle with respect to a direction of motion of the mobiledevice using the plurality of images, and means for storing themisalignment angle and the direction of motion in a storage device.

BRIEF DESCRIPTION OF THE DRAWINGS

The aforementioned features and advantages of the disclosure, as well asadditional features and advantages thereof, will be more clearlyunderstandable after reading detailed descriptions of embodiments of thedisclosure in conjunction with the following drawings.

FIG. 1A illustrates an example of misalignment angle with respect todirection of motion of a mobile device according to some aspects of thepresent disclosure. FIG. 1B illustrates a side view of the mobile device102 of FIG. 1A.

FIG. 2A-2C illustrate a method of performing perspective compensationfor camera pitch and asymmetric distribution of features according tosome aspects of the present disclosure.

FIG. 3A and FIG. 3B illustrate a method of detecting and removing movingobjects according to some aspects of the present disclosure.

FIG. 4 illustrates a block diagram of an exemplary mobile deviceaccording to some aspects of the present disclosure.

FIG. 5A illustrates a method of estimating speed and misalignment angleof a mobile device according to some aspects of the present disclosure.

FIG. 5B illustrates a method of estimating speed of a mobile deviceaccording to some aspects of the present disclosure.

FIG. 6 illustrates an exemplary method of determining positioncharacteristics of a mobile device according to some aspects of thepresent disclosure.

FIG. 7A illustrates a method of monitoring motion, misalignment angleand confidence associated with the misalignment angle for a mobiledevice over a period of time according to some aspects of the presentdisclosure.

FIG. 7B illustrates a graph showing changes to an average misalignmentangle of a mobile device over a period of time according to some aspectsof the present disclosure.

Like numbers are used throughout the figures.

DESCRIPTION OF EMBODIMENTS

Embodiments of methods and apparatuses for client-server based dynamicsearch are disclosed. The following descriptions are presented to enableany person skilled in the art to make and use the disclosure.Descriptions of specific embodiments and applications are provided onlyas examples. Various modifications and combinations of the examplesdescribed herein will be readily apparent to those skilled in the art,and the general principles defined herein may be applied to otherexamples and applications without departing from the spirit and scope ofthe disclosure. Thus, the present disclosure is not intended to belimited to the examples described and shown, but is to be accorded thewidest scope consistent with the principles and features disclosedherein. The word “exemplary” or “example” is used herein to mean“serving as an example, instance, or illustration.” Any aspect orembodiment described herein as “exemplary” or as an “example” in notnecessarily to be construed as preferred or advantageous over otheraspects or embodiments.

FIG. 1A illustrates an example of misalignment angle with respect todirection of motion of a mobile device according to some aspects of thepresent disclosure. As shown in FIG. 1A, in some use cases, direction ofmotion 101 of a mobile device 102 may be different from the mobiledevice orientation 103. The misalignment angle 105 is the angle betweenthe mobile device orientation 103 and the direction of motion 101. Notethat in some publications, the misalignment angle 105 may also bereferred to as an alignment angle. The ability to predict themisalignment angle 105 can be useful for pedestrian navigationapplications. The mobile device may include one or more camera(s) 108and a display 112

According to aspects of the present disclosure, when the mobile device102 is in pedestrian navigation mode, one or more cameras 108 of themobile device 102 may be configured to capture image frames fordetermining the misalignment angle 105. The image captured may be shownto a user in display 112.

FIG. 1B illustrates a side view of the mobile device 102 of FIG. 1A. Inone approach, the mobile device 102 may be configured to use eitherfront camera(s) 108 a (located in the front side of the mobile device)or back camera(s) 108 b (located in the back side of the mobile device)to capture image frames. For example, as shown in FIG. 1B, the frontcamera(s) 108 a can be configured to capture the field of view of anarea above the mobile device 102, and the back camera(s) 108 b can beconfigured to capture the field of view of an area below the mobiledevice 102. In another approach, the mobile device 102 may be configuredto use both the front camera(s) 108 a and the back camera(s) 108 b tocapture image frames in both front view and back view of the mobiledevice 102. With this approach, in some venues, features on the floor oron the ceiling may be gathered to estimate the direction of motion ofthe mobile device 102 with respect to the mobile device orientation 103.

In yet another approach, both of the front camera(s) 108 a and the backcamera(s) 108 b may be used in parallel. In this approach, errors causedby the two different perspectives in the front camera(s) 108 a and theback camera(s) 108 b may have opposite signs and may be compensatedbecause perspectives of the front camera(s) 108 a and the back camera(s)108 b are oriented 180 degrees apart from each other. In yet anotherapproach, either camera may be chosen based on which field of view hasmore features which are easier to track and based on which field of viewhas fewer moving objects.

There are numerous criteria may be used in choosing the front camera(s)108 a over the back camera(s) 108 b, or vice versa, including but notlimited to: 1) which field of view gives more features; 2) which fieldof view are easier to track; and 3) which field of view has fewer movingobjects. A camera may be chosen based on which one gives a higheraverage confidence metric for feature tracking. In addition, accordingto aspects of the present disclosure, the decision of which camera totrack can be made adaptively since the environment of the mobile device102 may change while it is being held by a user. In addition, accordingto aspects of the present disclosure, the mobile device 102 may beconfigured to use metrics to reject outliers since the image framesmight contain features of moving parts. For example, one source of suchmoving parts may be the feet of the user. Another source of such movingparts may be the head of the user.

FIG. 2A-2C illustrate a method of performing perspective compensationfor camera pitch and asymmetric distribution of features according tosome aspects of the present disclosure. As shown in FIG. 2A, the arrows(202 a, 204 a, 206 a and 208 a) illustrate an asymmetric distribution offeatures due to a possible camera pitch in a pedestrian navigationapplication. Such asymmetric distribution of features may lead toincorrect estimation for direction of motion of the mobile device 102.FIG. 2B illustrates the same asymmetric distribution of features(represented by the arrows) due to camera pitch in a pedestriannavigation application without the image background. The correspondingarrows are shown as 202 b, 204 b, 206 b and 208 b. FIG. 2C illustrateperspective adjusted distribution of features, represented by thecorresponding arrows 202 c, 204 c, 206 c, and 208 c.

To address the issues of asymmetric distribution of features, in oneapproach, the mobile device 102 may be configured to perform perspectivecorrection based on angle of the mobile device to vertical as calculatedusing sensors, such as magnetometer, accelerometer, and gyroscope. Inanother approach, the mobile device 102 may be configured to usefeatures near the center of an image frame in computing direction ofmotion of the mobile device 102. In yet another approach, a weightedaverage of features based on location, for example more weight forfeatures near center may be employed in computing direction of motion ofthe mobile device 102. For example, features along the center of animage, represented by arrow 202 a, may be assigned a weigh of 1 (100%),features represented by arrow 204 a may be assigned a weight of 0.8(80%), features represented by arrow 206 a may be assigned a weight of0.6 (60%), features represented by arrow 206 a may be assigned a weightof 0.4 (40%), and so on.

According to aspects of the present disclosure, identifying and trackingfeatures in image frames may be performed using a number of techniques.In one approach, a method of identifying features may be performed byexamining the minimum eigenvalue of each 2 by 2 gradient matrix. Thenthe features are tracked using a Newton-Raphson method of minimizing thedifference between the two windows. The method of multi-resolutiontracking allows for relatively large displacements between images. Notethat during tracking of features from one frame to the next frame,errors may accumulate. To detect potentially bad features, the mobiledevice 102 may be configured to monitor whether the image signal in thewindow around the feature in the current frame is still similar to theimage signal around the feature in the previous frame. Since featuresmay be tracked over many frames, the image content may be deformed. Toaddress this issue, consistency check may be performed with a similarityor an affine mapping.

According to aspects of the present disclosure, to identify an object inan image, points on the object may be extracted to provide featuredescriptions (also referred to as keypoints, feature points or featuresfor short) of the object. This description, extracted from a trainingimage, may then be used to identify the object when attempting to locatethe object in a test image containing many other objects. To performreliable recognition, the features extracted from the training image maybe detectable even under changes in image scale, noise and illumination.Such points usually lie on high-contrast regions of the image, such asobject edges.

Another characteristic of these features is that the relative positionsbetween them in the original scene may not change from one image toanother. For example, if only the four corners of a door are used asfeatures, they may work regardless of the door's position; but if pointsin the frame are used, the recognition may fail if the door is opened orclosed. Similarly, features located in articulated or flexible objectsmay typically not work if any change in their internal geometry happensbetween two images in the set being processed. In some implementations,SIFT detects and uses a larger number of features from the images, whichcan reduce the contribution of the errors caused by the local variationsin the average error of all feature matching errors. Thus, the disclosedmethod may identify objects even among clutter and under partialocclusion; because the SIFT feature descriptor can be invariant touniform scaling, orientation, and partially invariant to affinedistortion and illumination changes.

For example, keypoints of an object may first be extracted from a set ofreference images and stored in a database. An object is recognized in anew image by comparing each feature from the new image to this databaseand finding candidate matching features based on Euclidean distance oftheir feature vectors. From the full set of matches, subsets ofkeypoints that agree on the object and its location, scale, andorientation in the new image may be identified to filter out goodmatches. The determination of consistent clusters may be performed byusing a hash table implementation of a generalized Hough transform. Eachcluster of 3 or more features that agree on an object and its pose maythen be subject to further detailed model verification and subsequentlyoutliers may be discarded. The probability that a particular set offeatures indicates the presence of an object may then be computed basedon the accuracy of fit and number of probable false matches. Objectmatches that pass the tests can be identified as correct with highconfidence.

According to aspects of the present disclosure, image feature generationtransforms an image into a large collection of feature vectors, each ofwhich may be invariant to image translation, scaling, and rotation, aswell as invariant to illumination changes and robust to local geometricdistortion. These features share similar properties with neurons ininferior temporal cortex that are used for object recognition in primatevision. Key locations may be defined as maxima and minima of the resultof difference of Gaussians function applied in scale space to a seriesof smoothed and resampled images. Low contrast candidate points and edgeresponse points along an edge may be discarded. Dominant orientationsare assigned to localized keypoints. This approach ensures that thekeypoints are more stable for matching and recognition. SIFT descriptorsrobust to local affine distortion may then be obtained by consideringpixels around a radius of the key location, blurring and resampling oflocal image orientation planes.

Features matching and indexing may include storing SIFT keys andidentifying matching keys from the new image. In one approach, amodification of the k-d tree algorithm which is also referred to as thebest-bin-first search method that may be used to identify the nearestneighbors with high probability using a limited amount of computation.The best-bin-first algorithm uses a modified search ordering for the k-dtree algorithm so that bins in feature space may be searched in theorder of their closest distance from the query location. This searchorder requires the use of a heap-based priority queue for efficientdetermination of the search order. The best candidate match for eachkeypoint may be found by identifying its nearest neighbor in thedatabase of keypoints from training images. The nearest neighbors can bedefined as the keypoints with minimum Euclidean distance from the givendescriptor vector. The probability that a match is correct can bedetermined by taking the ratio of distance from the closest neighbor tothe distance of the second closest.

In one exemplary implementation, matches in which the distance ratio isgreater than 0.8 may be rejected, which eliminates 90% of the falsematches while discarding less than 5% of the correct matches. To furtherimprove the efficiency of the best-bin-first algorithm, search may becut off after checking a predetermined number (for example 100) nearestneighbor candidates. For a database of 100,000 keypoints, this mayprovide a speedup over exact nearest neighbor search by about 2 ordersof magnitude, yet results in less than a 5% loss in the number ofcorrect matches.

Note that with the exemplary implementation, the Hough Transform may beused to cluster reliable model hypotheses to search for keys that agreeupon a particular model pose. Hough transform may be used to identifyclusters of features with a consistent interpretation by using eachfeature to vote for object poses that may be consistent with thefeature. When clusters of features are found to vote for the same poseof an object, the probability of the interpretation being correct may behigher than for any single feature. An entry in a hash table may becreated to predict the model location, orientation, and scale from thematch hypothesis. The hash table can be searched to identify clusters ofat least 3 entries in a bin, and the bins may be sorted into decreasingorder of size.

According to aspects of the present disclosure, each of the SIFTkeypoints may specify 2D location, scale, and orientation. In addition,each matched keypoint in the database may have a record of itsparameters relative to the training image in which it is found. Thesimilarity transform implied by these 4 parameters may be anapproximation to the 6 degree-of-freedom pose space for a 3D object andalso does not account for any non-rigid deformations. Therefore, anexemplary implementation may use broad bin sizes of 30 degrees fororientation, a factor of 2 for scale, and 0.25 times the maximumprojected training image dimension (using the predicted scale) forlocation. The SIFT key samples generated at the larger scale may begiven twice the weight of those at the smaller scale. With thisapproach, the larger scale may in effect able to filter the most likelyneighbors for checking at the smaller scale. This approach also improvesrecognition performance by giving more weight to the least-noisy scale.According to aspects of the present disclosure, to avoid the issue ofboundary effects in bin assignment, each keypoint match may vote for the2 closest bins in each dimension, giving a total of 16 entries for eachhypothesis and further broadening the pose range.

FIG. 3A and FIG. 3B illustrate a method of detecting and removing movingobjects according to some aspects of the present disclosure. FIG. 3Aillustrates an image frame 302 that includes user's feet 304 a and 304 bas moving objects. FIG. 3B illustrates another image frame 306 thatincludes user's face 308 as a moving object. The user's feet 304 a and304 b and the user's face 308 may be considered as outliers; andfeatures of such outliers may be removed from the image frames capturesaccording to the methods described in the following sections.

According to aspects of the present disclosure, outliers may be removedby checking for agreement between each image feature and the model, fora given parameter solution. For example, given a linear least squaressolution, each match may be required to agree within half the errorrange that is used for the parameters in the Hough transform bins. Asoutliers are discarded, the linear least squares solution may beresolved with the remaining points, and the process may be iterated. Insome implementations, if less than a predetermined number of points(e.g. 3 points) remain after discarding outliers, the match may berejected. In addition, a top-down matching phase may be used to add anyfurther matches that agree with the projected model position, which mayhave been missed from the Hough transform bin due to the similaritytransform approximation or other errors.

The decision to accept or reject a model hypothesis can be based on adetailed probabilistic model. The method first computes an expectednumber of false matches to the model pose, given the projected size ofthe model, the number of features within the region, and the accuracy ofthe fit. A Bayesian probability analysis can then give the probabilitythat the object may be present based on the actual number of matchingfeatures found. A model may be accepted if the final probability for acorrect interpretation is greater than a predetermined percentage (forexample 95%).

According to aspects of the present disclosure, in one approach,rotation invariant feature transform (RIFT) method may be employed as arotation-invariant generalization of SIFT to address under clutter orpartial occlusion situations. The RIFT descriptor may be constructedusing circular normalized patches divided into concentric rings of equalwidth and within each ring a gradient orientation histogram may becomputed. To maintain rotation invariance, the orientation may bemeasured at each point relative to the direction pointing outward fromthe center.

In another approach, a generalized robust invariant feature (G-RIF)method may be used. The G-RIF encodes edge orientation, edge density andhue information in a unified form combining perceptual information withspatial encoding. The object recognition scheme uses neighboring contextbased voting to estimate object models.

In yet another approach, a speeded up robust feature (SURF) method maybe used which uses a scale and rotation-invariant interest pointdetector/descriptor that can outperform previously proposed schemes withrespect to repeatability, distinctiveness, and robustness. SURF relieson integral images for image convolutions to reduce computation time,and builds on the strengths of the leading existing detectors anddescriptors (using a fast Hessian matrix-based measure for the detectorand a distribution-based descriptor). The SURF method describes adistribution of Haar wavelet responses within the interest pointneighborhood. Integral images may be used for speed, and 64 dimensionsmay be used to reduce the time for feature computation and matching. Theindexing step may be based on the sign of the Laplacian, which increasesthe matching speed and the robustness of the descriptor.

In yet another approach, the principle component analysis SIFT(PCA-SIFT) method may be used. In some implementations, the PCA-SIFTdescriptor is a vector of image gradients in x and y direction computedwithin the support region. The gradient region can be sampled at 39×39locations. Thus, the vector can be of dimension 3042. The dimension canbe reduced to 36 with PCA. In yet another approach, the Gradientlocation-orientation histogram (GLOH) method can be employed, which isan extension of the SIFT descriptor designed to increase its robustnessand distinctiveness. In some implementations, the SIFT descriptor can becomputed for a log-polar location grid with three bins in radialdirection (the radius set to 6, 11, and 15) and 8 in angular direction,which results in 17 location bins. The central bin is not divided inangular directions. The gradient orientations may be quantized in 16bins resulting in 272 bin histogram. The size of this descriptor can bereduced with PCA. The covariance matrix for PCA can be estimated onimage patches collected from various images. The 128 largesteigenvectors may then be used for description.

In yet another approach, a two-object recognition algorithm may beemployed to use with the limitations of current mobile devices. Incontrast to the classic SIFT approach, the Features from AcceleratedSegment Test (FAST) corner detector can be used for feature detection.This approach distinguishes between the off-line preparation phase wherefeatures may be created at different scale levels and the on-line phasewhere features may be created at a current fixed scale level of themobile device's camera image. In one exemplary implementation, featuresmay be created from a predetermined fixed patch size (for example 15×15pixels) and form a SIFT descriptor with 36 dimensions. The approach canbe further extended by integrating a Scalable Vocabulary Tree in therecognition pipeline. This allows an efficient recognition of a largernumber of objects on mobile devices.

According to aspects of the present disclosure, the detection anddescription of local image features can help in object recognition. TheSIFT features can be local and based on the appearance of the object atparticular interest points, and may be invariant to image scale androtation. They may also be robust to changes in illumination, noise, andminor changes in viewpoint. In addition to these properties, thefeatures may be highly distinctive, relatively easy to extract and allowfor correct object identification with low probability of mismatch. Thefeatures can be relatively easy to match against a (large) database oflocal features, and generally probabilistic algorithms such ask-dimensional (k−d) trees with best-bin-first search may be used. Objectdescriptions by a set of SIFT features may also be robust to partialocclusion. For example, as few as 3 SIFT features from an object may besufficient to compute its location and pose. In some implementations,recognition may be performed in quasi real time, for small databases andon modern computer hardware.

According to aspects of the present disclosure, the random sampleconsensus (RANSAC) technique may be employed to remove outliers causedby moving objects in view of the camera. Note that the RANSAC uses aniterative method to estimate parameters of a mathematical model from aset of observed data which contains outliers. This method is anon-deterministic as it produces a reasonable result with an associatedprobability, where the probability may increase as more iteration isperformed.

In one exemplary implementation, a set of observed data values, aparameterized model which can be fitted to the observations withcorresponding confidence parameters. In this exemplary implementation,the method iteratively selects a random subset of the original data.These data can be hypothetical inliers and these hypothesis may then betested as follows:

-   -   1. A model is fitted to the hypothetical inliers, i.e. all free        parameters of the model are reconstructed from the inliers.    -   2. All other data are then tested against the fitted model and,        if a point fits well to the estimated model; it is considered as        a hypothetical inlier.    -   3. The estimated model can be considered acceptable if        sufficiently number of points have been classified as        hypothetical inliers.    -   4. The model is re-estimated from all hypothetical inliers,        because it has only been estimated from the initial set of        hypothetical inliers.    -   5. Finally, the model is evaluated by estimating the error of        the inliers relative to the model.

The above procedure can be repeated for a predetermined number of times,each time producing either a model which may be rejected because too fewpoints are classified as inliers or a refined model together with acorresponding error measure. In the latter case, the refined model iskept if the error is lower than the previously saved model.

In another exemplary implementation, moving objects in view of thecamera can be actively identified and removed using a model based motiontracking method. In one approach, the objective of tracking can betreated as a problem of model recognition. A binary representation ofthe target can be tracked, and a Hausdorff distance based search is usedto search regions of the image for the object. For a binaryrepresentation of the target (a model), output from the standard cannyedge detector of the Gaussian smoothed image is augmented with thenotion of a model history. At each frame, a Hausdorff search can beperformed on each target, using the canny edges from the current imageand the current model. In addition, an affine estimation may beperformed to approximate the net background motion. From the results ofthese two searches, information can be gathered about the target, and beused to approximate the motion of the target, as well as separate thebackground from motion in the region of the target. To be able to handlehazard/unusual conditions (such as the object becoming occluded goinginto a shadow, the object leaving the frame, or camera image distortionproviding bad image quality), history data about the target may beretained, such as the target's past motion and size change,characteristic views of the target (snapshots throughout time thatprovide an accurate representation of the different ways the target hasbeen tracked), and match qualities in the past.

The history of tracking the target can be useful in more than justaiding hazard/unusual conditions; that part of a solid motion trackingmethod can involve history data, and not just a frame by frame method ofmotion comparison. This history state can provide information regardinghow to decide what should be considered part of the target (e.g. thingsmoving close to the object moving at the same speed should beincorporated into the object), and with information about motion andsize, the method can predictively estimate where a lost object may havegone, or where it might reappear (which has been useful in recoveringtargets that leave the frame and reappear later in time).

An inherent challenge in the motion tracking method may be caused by thefact that the camera can have an arbitrary movement (as opposed to astationary camera), which makes developing a tracking system that canhandle unpredictable changes in camera motion difficult. Acomputationally efficient affine background estimation scheme may beused to provide information as to the motion of the camera and scene.

According to aspects of the present disclosure, an affine transformationfor the image can be performed at time t to the image at time t+dt,which allows correlating the motion in the two images. This backgroundinformation allows the method to synthesize an image at time t+dt fromthe image at time t and the affine transform that can be anapproximation of the net scene motion. This synthesized image can beuseful in generating new model information and removing backgroundclutter from the model space, because a difference of the actual imageat t+dt and the generated image at t+dt can be taken to remove imagefeatures from the space surrounding targets.

In addition to the use of the affine transform as a tool to clean-up thesearch space, it is also used to normalize the coordinate movement ofthe targets: by having a vector to track how the background may bemoving, and a vector to track how the target may be moving, a differenceof the two vector may be taken to generate a vector that describes themotion of the target with respect to the background. This vector allowsthe method to predictively match where the target should be, andanticipate hazard conditions (for example looking ahead in the directionof the motion can provide clues about upcoming obstacles, as well askeeping track of where the object may be in case of a hazard condition.When an object enters a hazard condition, the method may still be ableto estimate the background motion, and use that coupled with theknowledge of the model's previous movements to guess where the model mayreappear, or re-enter the frame.

The background estimation has been a key factor in the prolongedtracking of objects. Note that short term tracking may be performedwithout background estimation, but after a period of time, objectdistortion and hazards may be difficult to cope with effectively withouta good estimation of the background.

According to aspects of the present disclosure, one of the advantages ofusing the Hausdorff distance as a matching operator is that it can bequite tolerant of changes in shape during matching, but using theHausdorff distance as a matching operator may require the objects beingtracked be more accurately defined.

In one approach, straight dilation-based methods of grabbing a new modelfrom the time t+1 image can be used. Note that in some situations wherethere are non-object features close to the object (which occurs quiteoften), the dilation method may not be effective because it may slowlyincorporate the entire scene into the model. Thus, a method of updatingthe model from frame to frame that can be tolerant to changes in themodel shape, but not so relaxed that causing incorporating non-modelpixels into the model may be adopted. One exemplary implementation is touse a combination of background removal and adding the previous modelsto the current model match window and taking what seems to be stablepixels, as well as the new ones surrounding them, which over time mayeither get eliminated from the model because they may not be stable, orget incorporated into the model. This approach has been effective inkeeping the models relatively clean from clutter in the image. Forexample, with this approach, no longer does a road close to a truck getpulled into the model pixel by pixel. Note that the models may appear tobe dilated, but this may be a result of the history effect of how themodels are constructed, but it may also have the feature of making thesearch results more definite because this method can have more modelpixels to possibly match in the next frame.

Note that at each frame, there may be a significant amount ofcomputation to be performed. According to some implementations, themobile device can be configured to perform smoothing/feature extraction,Hausdorff matching each target (for example one match per model), aswell as affine background estimation. Each of these operations can bequite computationally expensive individually. In order to achievereal-time performance on a mobile device, the design can be configuredto use as much parallelism as possible.

FIG. 4 illustrates a block diagram of an exemplary mobile deviceaccording to some aspects of the present disclosure. The mobile device102 includes camera(s) 108 for capturing images of the environment,which may be either individual photos or frames of video. The mobiledevice 102 may also include sensors 116, which may be used to providedata with which the mobile device 102 can determine its position andorientation, i.e., pose. Examples of sensors that may be used with themobile device 102 include accelerometers, quartz sensors, gyros,micro-electromechanical system (MEMS) sensors used as linearaccelerometers, as well as magnetometers.

The mobile device 102 may also include a user interface 110 thatincludes display 112 capable of displaying images. The user interface110 may also include a keypad 114 or other input device through whichthe user can input information into the mobile device 102. If desired,the keypad 114 may be obviated by integrating a virtual keypad into thedisplay 112 with a touch sensor. The user interface 110 may also includea microphone 117 and one or more speakers 118, for example, if themobile device is a cellular telephone. Of course, mobile device 102 mayinclude other components unrelated to the present disclosure.

The mobile device 102 further includes a control unit 120 that isconnected to and communicates with the camera(s) 108 and sensors 116, aswell as the user interface 110, along with any other desired features.The control unit 120 may be provided by one or more processors 122 andassociated memory/storage 124. The control unit 120 may also includesoftware 126, as well as hardware 128, and firmware 130. The controlunit 120 includes a misalignment angle computation module 132 configuredto the misalignment angle between the orientation and the direction ofmotion of the mobile device 102. The control unit further includesmotion direction tracking module 133 configured to track the directionof motion of the mobile device 102 (which may indicate the direction ofmotion of the user). The control unit 120 further includes speedcomputation module 134 configured to compute the speed of the mobiledevice 102 (which may indicate the speed of the user). The misalignmentangle computation module 132, the motion direction tracking module 133,and the speed computation module 134 are illustrated separately fromprocessor 122 and/or hardware 128 for clarity, but may be combinedand/or implemented in the processor 122 and/or hardware 128 based oninstructions in the software 126 and the firmware 130.

FIG. 5A illustrates a method of estimating speed and misalignment angleof a mobile device according to some aspects of the present disclosure.In the example shown in FIG. 5A, the method starts in block 502, andthereafter moves to block 504 where the method collects sensor data.According to aspects of the present disclosure, the sensor data mayinclude image frames captured by one or more cameras 108 a and 108 b asdescribed in FIG. 1B. The sensor data may also include data collected byaccelerometer, gyroscope, magnetometer, and/or by a wireless receiver ofthe mobile device 102. In block 506, the method extracts features fromimage frames captured. In block 508, the method performs perspectivecompensation. According to aspects of the present disclosure, examplesof performing perspective compensation are provided in association withthe descriptions of FIG. 2A-FIG. 2C. In block 510, the method comparesfeatures between two consecutive image frames to determine separationsin pixels between the two consecutive image frames. According to aspectsof the present disclosure, methods of extracting and comparing featuresin image frames are provided in association with the descriptions ofFIG. 2A-FIG. 2C. In block 512, the method computes magnitude and angleof displacement based at least in part on the comparison of featuresbetween image frames performed in block 510.

In block 514, the method computes distance traversed between imageframes. In block 516, the method computes speed of the mobile deviceaccording to the methods described in association with FIG. 5B. In block518, the method determines direction of motion of the mobile devicebased at least in part on the angle of displacement determined in block512. In block 520, the method determines misalignment angle between theorientation and direction of motion of the mobile device based at leastin part on the direction of motion determined in block 518 and theorientation determined by data obtained by one or more sensors of themobile device 102. The method ends in block 520.

FIG. 5B illustrates a method of estimating speed of a mobile deviceaccording to some aspects of the present disclosure. In the exemplaryimplementation shown in FIG. 5B, by tracking features in image framescaptured by the one or more cameras 108 a and/or 108 b of the mobiledevice 102, the separation in pixels between two consecutive imageframes can be determined. The method converts a displacement in pixel inthe image frames to a displacement of distance in meters. Then, themethod determines the field of view of one or more cameras (108 a and/or108 b) and the height of the one or more cameras being held above theground. Note that due to perspective projection, the same distance cancorrespond to different number pixels in the image frames captured,depending on how far away an object may be from the perspective ofcamera of the mobile device. In some applications, a default height ofthe mobile device 102 being held above the ground may be estimated. Insome other applications, the height may be determined from the GPSlocation or from other positioning estimates (such as WiFi, Bluetoothmeasurements) of the mobile device 102.

As shown in FIG. 5B, for a lens 109 of a camera 108 projecting arectilinear image, the angle of view Alpha (α) can be calculated fromthe chosen dimension (d), and effective focal length (F) asAlpha=2*arctan(d/2F), where d represents the size of the sensor in thedirection measured. Note that the object image may occupy entire sensorif object angular size as seen from camera equals camera angle of view,with such physical object size being d*S1/S2.

Denoting number of pixels in the d dimension as MaxPix (that can be, forexample 720 or 1080 if images are obtained in Video mode, one pixel ofimage can correspond to d*S1/S2/MaxPix meters of the object (surface).If camera (in user hand) is traversing a stationary object and objectfeatures have moved by N pixels, it means that the camera has moved byNpix*d*S1/S2/MaxPix. If the movement has occurred over time interval T(like 1/30 second between two adjoining 30 Hz frames) then the speed ofthe mobile device can be estimated as Speed=Npix*d*S₁/S₂/MaxPix/T.

The above formula can be also presented in the alternative form withoutreference to the sensor size and distance from sensor to lens. Since inthe exemplary implementation, S1 represents distance of the camera fromthe floor H, and S1 is approximately equal F, the expression can bere-written as Speed=Npix*d*H/F/MaxPix/T. In addition,Alpha=2*arctan(d/2F) can be approximated as Alpha=d/F, which yieldsformula

Speed=Alpha/MaxPix*Npix*H/T.

In this formula Alpha/MaxPix includes a combination of known cameraparameters, object image displacement Npix and travel time T can bemeasured, and distance of the camera from the floor H can be eitherassumed or calibrated when the user speed can be known, for example fromGNSS or WiFi location.

FIG. 6 illustrates an exemplary method of determining positioncharacteristics of a mobile device according to some aspects of thepresent disclosure. In block 602, the control unit 120 can be configuredto capture a plurality of images that represent views from the mobiledevice. In block 604, the control unit 120 can be configured to adjustperspectives of the plurality of images based at least in part on anorientation of the mobile device. In block 606, the control unit 120 canbe configured to determine a misalignment angle with respect to adirection of motion of the mobile device using the plurality of images.In block 608, the control unit 120 can be configured to store themisalignment angle and the direction of motion in a storage device.

According to embodiments of the present disclosure, the methodsperformed in block 604 may further include methods performed in block610. In block 610, the control unit 120 can be configured to adjustperspectives of the plurality of images based on the orientation of themobile device calculated using data collected from one or more sensors,compensate for perspectives of the plurality of images using an areanear centers of the plurality of images, and/or compensate forperspectives of the plurality of images based at least in part on aweighted average of locations of features in the plurality of images.

According to embodiments of the present disclosure, the methodsperformed in block 606 may further include methods performed in blocks612 to 622. In block 612, the control unit 120 can be configured totrack features from the plurality of images, estimate direction ofmotion of the mobile device, estimate the orientation of the mobiledevice using sensor data, and determine the misalignment angle based atleast in part on the direction of motion and the orientation of themobile device.

In block 614, the control unit 120 can be configured to perform at leastone of: determine a confidence of the misalignment angle with respect tothe direction of motion of the mobile device using information providedby a gyroscope of the mobile device, determine the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using information provided by a magnetometer of the mobiledevice, and determine the confidence of the misalignment angle withrespect to the direction of motion of the mobile device using featuresof the plurality of images.

In block 616, the control unit 120 can be configured to determine aspeed estimation of the mobile device, determine a confidence of thespeed estimation of the mobile device, and apply the speed estimationand the confidence of the speed estimation to navigate a user of themobile device. In block 618, the control unit 120 can be configured toapply the misalignment angle and a confidence of the misalignment angleto navigate a user of the mobile device.

According to embodiments of the present disclosure, the methodsperformed in block 612 may further include methods performed in block620. In block 620, the control unit 120 can be configured to rejectoutliers in features of the plurality of images to eliminate at leastone moving object in the plurality of images.

According to embodiments of the present disclosure, the methodsperformed in block 616 may further include methods performed in block622. In block 622, the control unit 120 can be configured to extractfeatures from the plurality of images, compute an average displacementof the mobile device using the features from the plurality of images,and compute the speed estimate based at least in part on the averagedisplacement of the mobile device.

In block 622, the control unit 120 can be further configured to comparethe features from the plurality of images, determine a separation ofpixels between two consecutive images in the plurality of images,determine a time interval between the two consecutive images, andcalculate the speed estimate of the mobile device in accordance with theseparation of pixels and the time interval between the two consecutiveimages. In addition, the control unit 120 can be further configured tocalibrate a height of the mobile device according to at least one of GPSlocation information and WIFI location information of the mobile device.

According to some aspects of the present disclosure, the functionsdescribed in FIG. 5A and FIG. 6 may be implemented by the control unit120 of FIG. 4, potentially in combination with one or more otherelements. In some implementations, the functions may be performed byprocessor 122, software 126, hardware 128, and firmware 130 or acombination of the above to perform various functions of the apparatusdescribed in the present disclosure. In yet some other implementations,the functions described in FIG. 5A and FIG. 6 may be implemented by theprocessor 122 in combination with one or more other elements, forexample elements 108-118 and 124-134 of FIG. 4.

FIG. 7A illustrates a method of monitoring motion, misalignment angleand confidence associated with the misalignment angle for a mobiledevice over a period of time according to some aspects of the presentdisclosure. As shown in FIG. 7A, a first graph 702 shows themisalignment angle (also referred to as angle in FIG. 7A) between theorientation and the direction of motion of the mobile device 102 over aperiod of time, which is 80 seconds in this example. A second graph 704shows the corresponding confidence value associated with themisalignment angle over the same period of time. A third graph 706 showswhether the mobile device 102 is in motion over the same period of time.

In some applications, the confidence estimate can be an input to anavigation algorithm which can be configured to tell whether the mobiledevice may rely on the misalignment angle determined at a point in time.As shown above, the confidence estimate may be derived from the featuretracking algorithm. In some applications, the confidence value can alsobe estimated by the gyroscope and magnetometer. For example, thegyroscope and magnetometer may be configured to indicate the confidencevalue of the misalignment angle may be reduced when the user is turning(i.e. changing direction of motion).

In some other applications, when the gyroscope value exceeds athreshold, it can be inferred that the user may be either moving themobile device or turning; and in these cases, the confidence value maybe reduced accordingly. When the gyroscope indicates a turn, the filterthat computes averages of the misalignment angle may be reset. Note thatthe confidence metric illustrates how well features are being tracked.If the confidence value is low, the mobile device may be configured touse the angle from a previous time or from a different camera, such asswitching from the front-facing camera to the back-facing camera or viceversa.

FIG. 7B illustrates a graph showing changes to an average misalignmentangle of a mobile device over a period of time according to some aspectsof the present disclosure. As shown in FIG. 7B, in a first period 708from approximately 2 to 46 seconds, the average misalignment angle (alsoreferred to as average angle in FIG. 7B) is approximately −86 degrees.In a second period 710 from approximately 47 to 60 seconds, the averagemisalignment angle changes sharply, which typically indicates the usermay be moving the mobile device to a different position or the user maybe turning. In a third period 712 from approximately 61 to 75 seconds,the average misalignment angle is approximately −60 degrees. Suchinformation may be used to assist positioning and pedestrian navigationapplications.

According to aspects of the present disclosure, the method of monitoringmisalignment angle and direction of motion can be alternatively oradditionally performed as follows. In some implementations, for asensor-aided pedestrian navigation application running on the mobiledevice 102, the misalignment angle computation module 132 may beutilized to determine the angular offset between the orientation of themobile device 102 and the direction of forward motion of the mobiledevice 102, as given by the motion direction tracking module 133. Forexample, as shown in FIG. 1A, the misalignment angle can be defined bythe angular difference between the direction of motion M of a mobiledevice 102 and the direction of orientation O of the mobile device. Bycalculating and utilizing the misalignment angle, the direction ofmotion M of the mobile device 102 can be obtained in cases in whichconventional motion direction techniques fail. More particularly, themisalignment angle can have a wide range of value (e.g., from 0 to 360degrees) depending on the direction of orientation O of the mobiledevice 102. Without the misalignment angle even approximate conversionof device heading to motion direction can be challenging.

The misalignment angle is utilized to facilitate positioning of themobile device 102. For example, a mobile device 102 can be equipped witha compass or other mechanisms to provide information indicating theheading of the mobile device 102, which can be defined as the directionat which the mobile device is oriented (e.g., in relation to magneticnorth) within a given precision or tolerance amount. However, unless themobile device 102 is immovably positioned such that it is alwaysoriented in the direction of motion, the compass heading of the mobiledevice 102 alone does not represent the direction in which the mobiledevice 102 is moved. Thus, the misalignment angle can be utilized toconvert the direction of orientation of the mobile device 102 to thedirection of motion in the event that the mobile device 102 is notoriented in the direction of motion. As an example, the direction ofmotion in a compass-aided dead reckoning application can be computed asthe compass heading plus the misalignment angle.

The motion direction tracking module 133 and the misalignment anglecomputation module 132 can operate based on sensor data, informationobtained from a step detector (not shown), etc., to determine themisalignment angle associated with movement of a mobile device 102 beingcarried by a pedestrian. Initially, based on data collected fromaccelerometer(s) and/or the step detector, pedestrian steps can beidentified and the direction of gravity relative to the sensor axes ofthe mobile device 102 can be determined. These initial computations forma basis for the operation of the motion direction tracking module 133and the misalignment angle computation module 132, as described below.

With regard to pedestrian motion, such as walking, running, etc., thedirection of motion changes within a given pedestrian step and betweenconsecutive steps based on the biomechanics of pedestrian motion. Forexample, rather than proceeding in a constant forward direction, amoving pedestrian shifts left to right (e.g., left during a step withthe left foot and right during a step with the right foot) withsuccessive steps and vertically (e.g., up and down) within each step.Accordingly, transverse (lateral) acceleration associated with a seriesof pedestrian steps cycles between left and right with a two-step periodwhile forward and vertical acceleration cycle with a one-step period.

According to aspects of the present disclosure, the motion directiontracking module 133 may include a step shifter, a step summation moduleand a step correlation module (not shown in FIG. 4). The motiondirection tracking module 133 can leverage the above properties ofpedestrian motion to isolate the forward component of motion from thevertical and transverse components. For example, the motion directiontracking module 133 records acceleration information obtained fromaccelerometer(s) 28 (e.g., in a buffer) over consecutive steps. Torectify forward acceleration and suppress or cancel the transversecomponent of the acceleration, the motion direction tracking module 133utilizes the step shifter and the step summation module to sum odd andeven steps. In other words, the step shifter shifts acceleration datacorresponding to a series of pedestrian steps in time by one step.Subsequently, the step summation module sums the original accelerationinformation with the shifted acceleration information. As noted above,transverse changes sign with consecutive steps with a two-step perioddue to body rotation and rolling while forward and vertical accelerationexhibit a one-step period. As a result, summing pedestrian steps after aone-step shift reduces transverse acceleration while having minimalimpact on vertical or forward acceleration.

If the mobile device 102 is not centrally positioned on a pedestrian'sbody or shifts orientation during the pedestrian motion, transverseacceleration may not be symmetrical from step to step. Accordingly,while the step shifter and step summation module operate to reduce thetransverse component of acceleration, these modules may notsubstantially eliminate the transverse acceleration. To enhance theremoval of transverse acceleration, the step correlation module canfurther operate on the acceleration data obtained from theaccelerometer(s).

As a pedestrian steps forward (e.g., when walking), the center ofgravity of the pedestrian moves up at the beginning of the step and downat the end of the step. Similarly, the forward speed of the pedestriandecreases when the foot of the pedestrian reaches the ground at the endof a step and increases during the step. This relationship betweenforward and vertical motion during the progression of a pedestrian stepmay be leveraged by the step correlation module in further cancelingtransverse acceleration. In particular, if the acceleration associatedwith a pedestrian step is viewed as a periodic function, it can beobserved that the vertical acceleration and forward accelerationassociated with the step are offset by approximately a quarter of a step(e.g., 90 degrees). Accordingly, the step correlation module correlatesvertical acceleration with horizontal acceleration shifted (by the stepshifter) by one quarter step both forwards and backwards (e.g., +/−90degrees).

After shifting and correlation as described above, the vertical/forwardcorrelation may be comparatively strong due to the biomechanics ofpedestrian motion, while the vertical/transverse correlation may beapproximately zero. Thus, the correlations between vertical andhorizontal acceleration shifted forward and backward by one quarter stepare computed, and the forward shifted result may be subtracted from thebackward shifted result (since the results of the two correlations areopposite in sign) to further reduce the transverse component ofacceleration.

After the motion direction tracking module 133 substantially cancelstransverse acceleration as discussed above, the misalignment anglecomputation module 132 determines the angle between the forwardcomponent of acceleration and the orientation of the mobile device 102.According to aspects of the present disclosure, the misalignment anglecomputation module 132 may include an Eigen analysis module and an angledirection inference module (not shown in FIG. 4). In one exemplaryimplementation, the misalignment angle computation module 132 identifiesthe misalignment angle via Eigen analysis, as performed by an Eigenanalysis module, and further processing performed by an angle directioninference module. Based on information provided by the motion directiontracking module 133, the Eigen analysis module determines theorientation of the sensor axes of the mobile device with respect to theearth, from which a line corresponding to the direction of motion of themobile device 102 is obtained. The angle direction inference moduleanalyzes the obtained line, as well as forward and vertical accelerationdata corresponding to the corresponding pedestrian step(s), to determinethe direction of the misalignment angle based on the direction of motionof the mobile device 102 (e.g., forward or backward along the obtainedline). By doing so, the angle direction inference module operates toresolve forward/backward ambiguity associated with the misalignmentangle.

According to aspects of the present disclosure, the angle directioninference module leverages the motion signature of a pedestrian step todetermine the direction of the misalignment angle. As discussed above,forward and vertical acceleration corresponding to a pedestrian step arerelated due to the mechanics of leg rotation, body movement, and otherfactors associated with pedestrian motion. Thus, the angle directioninference module can be configured to utilize knowledge of theserelationships to identify whether a motion direction is forward orbackward along a given line.

While the above discussion relates to obtaining a two-dimensional motiondirection, e.g., with respect to a horizontal plane, similar techniquesmay be utilized to obtain a direction of motion in three dimensions.Thus, the techniques described herein can be extended to account forchanges in altitude, pedestrian motion along an uneven surface, and/orother factors impacting the direction of motion in three dimensions.

Additionally, the techniques described above can be extended to leveragea gyroscope in addition to accelerometer(s). With further reference tothe biomechanics of pedestrian motion, leg rotation and other associatedmovements during a pedestrian step can be classified as angularmovements, e.g., measured in terms of pitch or roll. Accordingly, agyroscope can be used to separate gravity from acceleration due tomovement such that the reference frame for computation can be rotated toaccount for the orientation of the mobile device 102 prior to thecalculations described above.

Note that at least paragraphs [0097]-[0099], FIG. 4, FIG. 5A, FIG. 5B,FIG. 6 and their corresponding descriptions provide means for capturinga plurality of images that represent views from the mobile device, meansfor adjusting perspectives of the plurality of images based at least inpart on an orientation of the mobile device, means for determining amisalignment angle with respect to a direction of motion of the mobiledevice using the plurality of images, and means for storing themisalignment angle and the direction of motion in a storage device. Atleast paragraphs [0097]-[0099], FIG. 1A, FIG. 1B, FIG. 4, FIG. 5A, FIG.5B, FIG. 6 and their corresponding descriptions provide means foradjusting perspectives of the plurality of images based on theorientation of the mobile device calculated using data collected fromone or more sensors; means for compensating for perspectives of theplurality of images using an area near centers of the plurality ofimages; means for compensating for perspectives of the plurality ofimages based at least in part on a weighted average of locations offeatures in the plurality of images; means for tracking features fromthe plurality of images; means for estimating direction of motion of themobile device; means for estimating the orientation of the mobile deviceusing sensor data; means for determining the misalignment angle based atleast in part on the direction of motion and the orientation of themobile device; means for determining a confidence of the misalignmentangle with respect to the direction of motion of the mobile device usinginformation provided by a gyroscope of the mobile device; means fordetermining the confidence of the misalignment angle with respect to thedirection of motion of the mobile device using information provided by amagnetometer of the mobile device; means for determining the confidenceof the misalignment angle with respect to the direction of motion of themobile device using features of the plurality of images; means fordetermining a speed estimation of the mobile device; means fordetermining a confidence of the speed estimation of the mobile device;and means for applying the speed estimation and the confidence of thespeed estimation to navigate a user of the mobile device.

The methodologies and mobile device described herein can be implementedby various means depending upon the application. For example, thesemethodologies can be implemented in hardware, firmware, software, or acombination thereof. For a hardware implementation, the processing unitscan be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, electronic devices, other electronicunits designed to perform the functions described herein, or acombination thereof. Herein, the term “control logic” encompasses logicimplemented by software, hardware, firmware, or a combination.

For a firmware and/or software implementation, the methodologies can beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine readable mediumtangibly embodying instructions can be used in implementing themethodologies described herein. For example, software codes can bestored in a memory and executed by a processing unit. Memory can beimplemented within the processing unit or external to the processingunit. As used herein the term “memory” refers to any type of long term,short term, volatile, nonvolatile, or other storage devices and is notto be limited to any particular type of memory or number of memories, ortype of media upon which memory is stored.

If implemented in firmware and/or software, the functions may be storedas one or more instructions or code on a computer-readable medium.Examples include computer-readable media encoded with a data structureand computer-readable media encoded with a computer program.Computer-readable media may take the form of an article of manufacturer.Computer-readable media includes physical computer storage media and/orother non-transitory media. A storage medium may be any available mediumthat can be accessed by a computer. By way of example, and notlimitation, such computer-readable media can comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer; disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk and Blu-ray disc where disks usually reproducedata magnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

In addition to storage on computer readable medium, instructions and/ordata may be provided as signals on transmission media included in acommunication apparatus. For example, a communication apparatus mayinclude a transceiver having signals indicative of instructions anddata. The instructions and data are configured to cause one or moreprocessors to implement the functions outlined in the claims. That is,the communication apparatus includes transmission media with signalsindicative of information to perform disclosed functions. At a firsttime, the transmission media included in the communication apparatus mayinclude a first portion of the information to perform the disclosedfunctions, while at a second time the transmission media included in thecommunication apparatus may include a second portion of the informationto perform the disclosed functions.

The disclosure may be implemented in conjunction with various wirelesscommunication networks such as a wireless wide area network (WWAN), awireless local area network (WLAN), a wireless personal area network(WPAN), and so on. The terms “network” and “system” are often usedinterchangeably. The terms “position” and “location” are often usedinterchangeably. A WWAN may be a Code Division Multiple Access (CDMA)network, a Time Division Multiple Access (TDMA) network, a FrequencyDivision Multiple Access (FDMA) network, an Orthogonal FrequencyDivision Multiple Access (OFDMA) network, a Single-Carrier FrequencyDivision Multiple Access (SC-FDMA) network, a Long Term Evolution (LTE)network, a WiMAX (IEEE 802.16) network and so on. A CDMA network mayimplement one or more radio access technologies (RATs) such as cdma2000,Wideband-CDMA (W-CDMA), and so on. Cdma2000 includes IS-95, IS2000, andIS-856 standards. A TDMA network may implement Global System for MobileCommunications (GSM), Digital Advanced Mobile Phone System (D-AMPS), orsome other RAT. GSM and W-CDMA are described in documents from aconsortium named “3rd Generation Partnership Project” (3GPP). Cdma2000is described in documents from a consortium named “3rd GenerationPartnership Project 2” (3GPP2). 3GPP and 3GPP2 documents are publiclyavailable. A WLAN may be an IEEE 802.11x network, and a WPAN may be aBluetooth network, an IEEE 802.15x, or some other type of network. Thetechniques may also be implemented in conjunction with any combinationof WWAN, WLAN and/or WPAN.

A mobile station refers to a device such as a cellular or other wirelesscommunication device, personal communication system (PCS) device,personal navigation device (PND), Personal Information Manager (PIM),Personal Digital Assistant (PDA), laptop or other suitable mobile devicewhich is capable of receiving wireless communication and/or navigationsignals. The term “mobile station” is also intended to include deviceswhich communicate with a personal navigation device (PND), such as byshort-range wireless, infrared, wire line connection, or otherconnection—regardless of whether satellite signal reception, assistancedata reception, and/or position-related processing occurs at the deviceor at the PND. Also, “mobile station” is intended to include alldevices, including wireless communication devices, computers, laptops,etc. which are capable of communication with a server, such as via theInternet, Wi-Fi, or other network, and regardless of whether satellitesignal reception, assistance data reception, and/or position-relatedprocessing occurs at the device, at a server, or at another deviceassociated with the network. Any operable combination of the above arealso considered a “mobile station.”

Designation that something is “optimized,” “required” or otherdesignation does not indicate that the current disclosure applies onlyto systems that are optimized, or systems in which the “required”elements are present (or other limitation due to other designations).These designations refer only to the particular describedimplementation. Of course, many implementations are possible. Thetechniques can be used with protocols other than those discussed herein,including protocols that are in development or to be developed.

One skilled in the relevant art will recognize that many possiblemodifications and combinations of the disclosed embodiments may be used,while still employing the same basic underlying mechanisms andmethodologies. The foregoing description, for purposes of explanation,has been written with references to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the disclosure to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described to explain the principles of thedisclosure and their practical applications, and to enable othersskilled in the art to best utilize the disclosure and variousembodiments with various modifications as suited to the particular usecontemplated.

We claim:
 1. A method of determining position characteristics of amobile device, comprising: capturing a plurality of images thatrepresent views from the mobile device; adjusting perspectives of theplurality of images based at least in part on an orientation of themobile device; determining a misalignment angle with respect to adirection of motion of the mobile device using the plurality of images;and storing the misalignment angle and the direction of motion in astorage device.
 2. The method of claim 1, wherein adjusting perspectivesof the plurality of images comprises at least one of: adjustingperspectives of the plurality of images based on the orientation of themobile device calculated using data collected from one or more sensors;compensating for perspectives of the plurality of images using an areanear centers of the plurality of images; and compensating forperspectives of the plurality of images based at least in part on aweighted average of locations of features in the plurality of images. 3.The method of claim 1, wherein determining the misalignment anglecomprises: tracking features from the plurality of images; estimatingdirection of motion of the mobile device; estimating the orientation ofthe mobile device using sensor data; and determining the misalignmentangle based at least in part on the direction of motion and theorientation of the mobile device.
 4. The method of claim 3, wherein thetracking features from the plurality of images comprises: rejectingoutliers in features of the plurality of images to eliminate at leastone moving object in the plurality of images.
 5. The method of claim 1,further comprising at least one of: determining a confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using information provided by a gyroscope of the mobile device;determining the confidence of the misalignment angle with respect to thedirection of motion of the mobile device using information provided by amagnetometer of the mobile device; and determining the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using features of the plurality of images.
 6. The method of claim1 further comprises: determining a speed estimation of the mobiledevice; determining a confidence of the speed estimation of the mobiledevice; and applying the speed estimation and the confidence of thespeed estimation to navigate a user of the mobile device.
 7. The methodof claim 6, wherein determining a speed estimate comprises: extractingfeatures from the plurality of images; computing an average displacementof the mobile device using the features from the plurality of images;and computing the speed estimate based at least in part on the averagedisplacement of the mobile device.
 8. The method of claim 7, whereincomputing the speed estimate comprises: comparing the features from theplurality of images; determining a separation of pixels between twoconsecutive images in the plurality of images; determining a timeinterval between the two consecutive images; and calculating the speedestimate of the mobile device in accordance with the separation ofpixels and the time interval between the two consecutive images.
 9. Themethod of claim 8 further comprises: calibrating a height of the mobiledevice according to at least one of GPS location information and WIFIlocation information of the mobile device.
 10. The method of claim 1further comprises: applying the misalignment angle and a confidence ofthe misalignment angle to navigate a user of the mobile device.
 11. Anapparatus, comprising: a control unit including processing logic, theprocessing logic comprising: logic configured to capture a plurality ofimages that represent views from a mobile device; logic configured toadjust perspectives of the plurality of images based at least in part onan orientation of the mobile device; logic configured to determine amisalignment angle with respect to a direction of motion of the mobiledevice using the plurality of images; and logic configured to store themisalignment angle and the direction of motion in a storage device. 12.The apparatus of claim 11, wherein the logic configured to adjustperspectives of the plurality of images comprises at least one of: logicconfigured to adjust perspectives of the plurality of images based onthe orientation of the mobile device calculated using data collectedfrom one or more sensors; logic configured to compensate forperspectives of the plurality of images using an area near centers ofthe plurality of images; and logic configured to compensate forperspectives of the plurality of images based at least in part on aweighted average of locations of features in the plurality of images.13. The apparatus of claim 11, wherein the logic configured to determinethe misalignment angle comprises: logic configured to track featuresfrom the plurality of images; logic configured to estimate direction ofmotion of the mobile device; logic configured to estimate theorientation of the mobile device using sensor data; and logic configuredto determine the misalignment angle based at least in part on thedirection of motion and the orientation of the mobile device.
 14. Theapparatus of claim 13, wherein the logic configured to track featuresfrom the plurality of images comprises: logic configured to rejectoutliers in features of the plurality of images to eliminate at leastone moving object in the plurality of images.
 15. The apparatus of claim11, further comprising at least one of: logic configured to determine aconfidence of the misalignment angle with respect to the direction ofmotion of the mobile device using information provided by a gyroscope ofthe mobile device; logic configured to determine the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using information provided by a magnetometer of the mobiledevice; and logic configured to determine the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using features of the plurality of images.
 16. The apparatus ofclaim 11 further comprises: logic configured to determine a speedestimation of the mobile device; logic configured to determine aconfidence of the speed estimation of the mobile device; and logicconfigured to apply the speed estimation and the confidence of the speedestimation to navigate a user of the mobile device.
 17. The apparatus ofclaim 16, wherein the logic configured to determine a speed estimatecomprises: logic configured to extract features from the plurality ofimages; logic configured to compute an average displacement of themobile device using the features from the plurality of images; and logicconfigured to compute the speed estimate based at least in part on theaverage displacement of the mobile device.
 18. The apparatus of claim17, wherein the logic configured to compute the speed estimatecomprises: logic configured to compare the features from the pluralityof images; logic configured to determine a separation of pixels betweentwo consecutive images in the plurality of images; logic configured todetermine a time interval between the two consecutive images; and logicconfigured to calculate the speed estimate of the mobile device inaccordance with the separation of pixels and the time interval betweenthe two consecutive images.
 19. The apparatus of claim 18 furthercomprises: logic configured to calibrate a height of the mobile deviceaccording to at least one of GPS location information and WIFI locationinformation of the mobile device.
 20. The apparatus of claim 11 furthercomprises: logic configured to apply the misalignment angle and aconfidence of the misalignment angle to navigate a user of the mobiledevice.
 21. A computer program product comprising non-transitory mediumstoring instructions for execution by one or more computer systems, theinstructions comprising: instructions for capturing a plurality ofimages that represent views from a mobile device; instructions foradjusting perspectives of the plurality of images based at least in parton an orientation of the mobile device; instructions for determining amisalignment angle with respect to a direction of motion of the mobiledevice using the plurality of images; and instructions for storing themisalignment angle and the direction of motion in a storage device. 22.The computer program product of claim 21, wherein the instructions foradjusting perspectives of the plurality of images comprises at least oneof: instructions for adjusting perspectives of the plurality of imagesbased on the orientation of the mobile device calculated using datacollected from one or more sensors; instructions for compensating forperspectives of the plurality of images using an area near centers ofthe plurality of images; and instructions for compensating forperspectives of the plurality of images based at least in part on aweighted average of locations of features in the plurality of images.23. The computer program product of claim 21, wherein the instructionsfor determining the misalignment angle comprises: instructions fortracking features from the plurality of images; instructions forestimating direction of motion of the mobile device; instructions forestimating the orientation of the mobile device using sensor data; andinstructions for determining the misalignment angle based at least inpart on the direction of motion and the orientation of the mobiledevice.
 24. The computer program product of claim 23, wherein theinstructions for tracking features from the plurality of imagescomprises: instructions for rejecting outliers in features of theplurality of images to eliminate at least one moving object in theplurality of images.
 25. The computer program product of claim 21,further comprising at least one of: instructions for determining aconfidence of the misalignment angle with respect to the direction ofmotion of the mobile device using information provided by a gyroscope ofthe mobile device; instructions for determining the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using information provided by a magnetometer of the mobiledevice; and instructions for determining the confidence of themisalignment angle with respect to the direction of motion of the mobiledevice using features of the plurality of images.
 26. The computerprogram product of claim 21 further comprises: instructions fordetermining a speed estimation of the mobile device; instructions fordetermining a confidence of the speed estimation of the mobile device;and instructions for applying the speed estimation and the confidence ofthe speed estimation to navigate a user of the mobile device.
 27. Thecomputer program product of claim 26, wherein the instructions fordetermining a speed estimate comprises: instructions for extractingfeatures from the plurality of images; instructions for computing anaverage displacement of the mobile device using the features from theplurality of images; and instructions for computing the speed estimatebased at least in part on the average displacement of the mobile device.28. The computer program product of claim 27, wherein the instructionsfor computing the speed estimate comprises: instructions for comparingthe features from the plurality of images; instructions for determininga separation of pixels between two consecutive images in the pluralityof images; instructions for determining a time interval between the twoconsecutive images; and instructions for calculating the speed estimateof the mobile device in accordance with the separation of pixels and thetime interval between the two consecutive images.
 29. The computerprogram product of claim 28 further comprises: instructions forcalibrating a height of the mobile device according to at least one ofGPS location information and WIFI location information of the mobiledevice.
 30. The computer program product of claim 21 further comprises:instructions for applying the misalignment angle and a confidence of themisalignment angle to navigate a user of the mobile device.
 31. Asystem, comprising: means for capturing a plurality of images thatrepresent views from a mobile device; means for adjusting perspectivesof the plurality of images based at least in part on an orientation ofthe mobile device; means for determining a misalignment angle withrespect to a direction of motion of the mobile device using theplurality of images; and means for storing the misalignment angle andthe direction of motion in a storage device.
 32. The system of claim 31,wherein the means for adjusting perspectives of the plurality of imagescomprises at least one of: means for adjusting perspectives of theplurality of images based on the orientation of the mobile devicecalculated using data collected from one or more sensors; means forcompensating for perspectives of the plurality of images using an areanear centers of the plurality of images; and means for compensating forperspectives of the plurality of images based at least in part on aweighted average of locations of features in the plurality of images.33. The system of claim 31, wherein the means for determining themisalignment angle comprises: means for tracking features from theplurality of images; means for estimating direction of motion of themobile device; means for estimating the orientation of the mobile deviceusing sensor data; and means for determining the misalignment anglebased at least in part on the direction of motion and the orientation ofthe mobile device.
 34. The system of claim 31, further comprising atleast one of: means for determining a confidence of the misalignmentangle with respect to the direction of motion of the mobile device usinginformation provided by a gyroscope of the mobile device; means fordetermining the confidence of the misalignment angle with respect to thedirection of motion of the mobile device using information provided by amagnetometer of the mobile device; and means for determining theconfidence of the misalignment angle with respect to the direction ofmotion of the mobile device using features of the plurality of images.35. The system of claim 31 further comprises: means for determining aspeed estimation of the mobile device; means for determining aconfidence of the speed estimation of the mobile device; and means forapplying the speed estimation and the confidence of the speed estimationto navigate a user of the mobile device.